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ABSTRACT 



The evaluation conduct factor involves matters such as 



whether correct methodology is being used, whether the measurements used are 
philosophically consistent with the goals of the program, timeliness, and 
format of the evaluation report, and whether recommendations provided in the 
report are usable. This study analyzed the conduct factor in the use of 
student data for decision making in Texas bilingual education programs. 
Surveys were completed by 93 directors of bilingual education programs, and 
survey data were merged with district data from the states information 
management system. Texas bilingual program directors reported that they do 
use data to improve education practices within their school districts. 
However, they do not use the wealth of data collected for reports to the 
state, but concentrate on a single component, Texas Assessment of Academic 
Skills (TAAS) data. Examination of district practices shows that the use of 
only one source of data is not a sound educational practice, but the valuing 
of TAAS data over any other data type may actually be quite rational because 
these data are the only ones used by the state to determine school ratings . 
Survey results do show the importance of three components of the conduct 
factor: (1) a strong belief in using Spanish for instruction associated with 

higher levels of use of Spanish norm-referenced testing data; (2) correlation 
between data use and data availability; and (3) the great emphasis on TAAS 
data. (Contains 27 references.) (SLD) 
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Use of accountability data for decision-making in Texas bilingual education programs: 

An examination of the conduct factor 
Introduction 

During the past decade Texas public schools have implemented several accountability 
initiatives, incurring significant cost in terms of human and financial resources. A primary component 
of these accountability initiatives is data pertaining to student academic performance. For example, 
students participate in the Texas Assessment of Academic Skills, Texas Primary Reading Inventory, 
end-of-course tests, and in some districts, norm-reference tests. In addition, the State requires schools 
to collect data on student attendance, dropout, and graduation rates. Together these pieces of 
information are used by the State in its Academic Excellence Indicator System (AEIS), which ranks 
and rewards schools and districts according to specified standards of performance. The data source 
for AEIS is the Public Education Information Management System (PEIMS). 

The Texas Education Agency (TEA) states, “The goal of the Academic Excellence 
Indicator System (AEIS) assessment program in Texas is to measure student progress toward 
achieving academic excellence” (on-line http://www.tea.state.tx.us/student.assessment/techdig/ 
chaptl.htm) . Elsewhere TEA states, “One of the basic goals of PEIMS, as adopted by the State 
Board of Education in 1986, is to improve education practices of local school districts” (online 
http://www.tea.state.tx.us/perfreport/aeis/about.aeis.html) . Thus, an expected outcome of these 
accountability systems is to provide decision-makers information for the purposes of assessing and 
improving educational practices in Texas schools. 

Informed decision-making and improved educational practice are two key purposes 
advanced by the program evaluation movement (Patton, 1997). Weiss (1967), in a seminal paper on 
evaluation utilization, explains 
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The basic rationale for evaluation is that it provides information for action. Its primary 
justification is that it contributes to the rationalization of decision-making. Although it can 
serve such other functions as knowledge-building and theory-testing, unless it gains serious 
hearing when program decisions are made, it fails in its major purpose, (p. 318) 

Alkin, Daillak , and White (1979) reiterate these purposes, “evaluation is distinctly practical 
and utilitarian, and to succeed, or to ‘work,’ it must have influence upon the decision-making 
process in its target programs” (p. 14). However, the literature on program evaluation is saturated 
with references to the difficulties of achieving these purposes, which may appear on the surface to be 
both simple and sensible (Cook, Leviton, & Shadish, 1985), but are made complicated by political, 
social, cultural, and economical factors among others. 

Indeed, so concerned were scholars with the lack of utilization of evaluation results in 
decision making that calls were made for the “systematic study of conditions associated with 
utilization of evaluation results” (Weiss, 1972, p. 321). ). In a related course, Alkin and Stecher 
(1983) asked that further studies examine the difference in types of decisions made with evaluation 
data. And even more recent is Alkin’s (1990) call for studies on the characteristics of those who use 
and disseminate information. 

Research that was initated to address these calls suggest that contrary to the beliefs of the 
mid-70’s, when it was thought that the only necessary ingredient for an effective evaluation was 
accurate and reliable data, it is now known that many factors affect the extent of evaluation use. 
Burry, et.al. (1984) identified three key factors that contribute to program evaluation use: the 
human factor, the evaluation context factor, and the evaluation procedures or conduct factor. The 
human factor includes matters such as the personal characteristics of the evaluation user, the 
user’s autonomy, whether or not the user perceives the data to be relevant to decisions being 
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made, and whether or not the user perceives the data as validating or threatening. The evaluation 
context factor relates to issues such as whether or not the program is new and whether or not the 
program is highly politicized. The evaluation conduct factor involves matters such as whether or 
not correct methodology is being used, whether or not the measurements used are 
philosophically consistent with the goals of the program, the timeliness and format of the 
evaluation report, and whether or not recommendations provided in the report are useable. 

Problem and Purpose of the Study 

While school, district, and state employees in Texas go about the task of administering, 
collecting, and reporting student assessment and other kinds of information prescribed by the State’s 
accountability systems, questions remain pertaining to the use of such data for the purposes of 
informing decision making and of improving educational practice. An extensive search of the 
literature reveals no study to date has been published that attempts to address this issue. Given the 
significant investment of public monies and human time required to carry out these accountability 
systems, a study focused on describing and analyzing the use of student data for such purposes is 
warranted. To narrow the scope of this investigation, this study is confined to an analysis of the 
conduct factor in the use of student data for decision-making in Texas bilingual education programs. 

Of particular interest to Texas educators is the academic progress of limited English 
proficient (LEP) students. These students, who comprise over 12% of the total population of Texas 
students, the majority of whom are Spanish speaking, come to school without the necessary skills to 
participate fully in general education programs. They are “at-risk” because of their lack of English 
skills and are often at-risk for additional factors, including having a low socio-economic status, lack 
of consistent schooling prior to coming to the U.S., and being overage in their classrooms 
(Rumberger & Larson, 1 998). Evidence from research in bilingual education suggests that students 
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typically enter school at or below the 20 th percentile of the general education population on norm- 
reference tests (Thomas & Collier, 1995). 

When students are already behind in school, time devoted to assessment is a significant 
investment of instructional time (Gandara & Merino, 1993). LEP students participate in an even 
greater number of required tests than their English proficient peers, including annual oral language 
proficiency tests, the Reading Proficiency Test in English, and often Spanish norm-reference and 
criterion-reference tests. In order to justify this testing, bilingual program directors must utilize the 
data for selecting appropriate program models and guiding the implementation of these models. In 
his 1 992, Review of Practices and Problems in the Evaluation of Bilingual Education, Lam 
concludes, “For the promotion of quality of educational and economic opportunities for LEP 
children, evaluation of bilingual education must be taken seriously for the instrumental purposes that 
it is designed to serve — program accountability and improvement”(p. 199). 

Leaders of educational programs may realize that sharing information within the district is 
critically important. However, accountability data may not have the intended impact on educational 
programs. For a variety of reasons, educators may not use such data in decision-making. This is 
double loss because not only are resources such as time and money wasted, instructional programs 
are not improved as a result (Thompson, 1994, p. 57). Therefore, the purpose of this study is to 
examine the nature of the conduct factor in the use of student accountability data by Texas bilingual 
education program directors to make decisions about the implementation of quality programs for 
limited English proficient students. 

The Conduct Factor and Texas Bilingual Education 

Burry, et.al., (1984) describes the conduct or procedures factor in evaluation use, as “the 
actual conduct of the evaluation . . . how the evaluator and users work together, the procedures 
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used in the evaluation, and the quality of the information it provides” (p. 15). Building on the 
explanation of the earlier factors, context and human, Burry writes that evaluation procedures 
and the perceptions of the methodology also affect use. The educational leader must believe that 
the appropriate evaluation procedures have been used to gather and analyze data before the 
results of evaluations will be used in decision making. 

In considering this factor as it relates to bilingual education, bilingual program directors 
may have well-founded doubts that assessment data are a result of valid and reliable procedures. 
Lam (1992) writes that “although data have been accumulated for many years, the poor quality 
of the evaluation efforts have severely hampered attempts to draw conclusions about the impact 
of educational instruction designed to serve LEP students” (p. 182). Currently, guidelines are 
available that can improve efforts to conduct methodologically sound evaluations (Berk & Rossi, 
1999, Thinking about program evaluation , 2 nd ed.\ Boulmetis & Dutwin, 2000, The ABCs of 
Evaluation', Chelimsky & Sadish, 1997, Evaluation for the 21 s ' century: A handbook). However, 
few resources target the evaluation of programs for limited English proficient (LEP) students in 
second language programs. 

Although validity of the assessment data, or perceptions of its validity, may be a major 

obsticle for educators, it is only one component of a well-developed program evaluation. 

\ 

Rumberger and Larson (1988) have identified a number of key indicators that should be 
considered in the evaluation of programs for LEP students in addition to standardized testing 
data. These indicators include: classroom grades, attendance information, student discipline 
information, student mobility information, student status of being “on track to graduate,” 
retention rates, student study habits as reported on report cards, and student socioeconomic 
status. Thus, besides using assessment data, bilingual program directors should use other types of 
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data to inform program selection, implementation, monitoring, and improvement. These 
indicators provide a more comprehensive analysis of student progress in second language 
programs. 

Dickey (1980) reports that instructional leaders who do not use evaluation data often 
believe the evaluation procedures are inappropriate, using “inadequate instruments, procedures 
that were disruptive and time consuming and procedures that were philosophically inconsistent 
with the project goals” (p. 73). Thus, one could hypothesize that bilingual directors would report 
a higher level of usage for measurements that were consistent with their personal philosophy of 
education. Thus, the question arises: Do bilingual program directors who believe in using the 
student’s native language for instruction report higher levels of use of Spanish norm-referenced 
tests than bilingual program directors who believe in English immersion programs? 

In addition to beliefs about evaluation procedures, the presentation of evaluation data is a 
component of evaluation conduct that has received attention. The timeliness of the evaluation 
report, the overall quality of the report, and the treatment of report recommendations impact 
whether or not evaluation data will be used. Each of these elements will be addressed in the 
following paragraphs. 

First, the timing of the report is critical to whether or not the data will be used. “The 
timing of an evaluation can by itself restrict the potential utility by not meshing with the timing 
of the program planning (Alkin, 1985, p. 134). Although the data may have been useful if 
available on time, the data are not relevant after the decision has been made. Dickey (1980) 
asserts, “timely information for decision making should be the primary focus” (p. 76). Decision- 
makers must be able to identify the needed data early enough to request a timely report. Systems 
within the district must be in place to facilitate gathering and report the data to the decision 
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maker. Thus, the following question emerges concerning the timeliness of data according to 
bilingual program directors: Do bilingual program directors who report data are available to 
them when needed report higher levels of data usage than bilingual program directors who report 
the data are not available to them when needed? 

Finally, the overall quality of the evaluation report itself contributes to whether or not the 
data are used. Alkin, Kosecoff, Fitz-Gibbon, and Seligman (1974), found that “no matter how 
high the quality of an actual evaluation, it will be of little worth if the results are not well 
presented” (p. 17). Leviton and Hughes (1981) write, “The way evaluation is presented to users 
affects their comprehension and thus the extent of use” (p. 537). Educational decision makers 
have reported they prefer narrative data in oral and written reports (Leviton & Hughes, 1981; 
Thompson, 1994). Educators have also emphasized the importance of familiar report formats 
(Burry, et.al., 1984). Because the State of Texas has been implementing the TAAS test for 
several years, Texas educators are now familiar with the format of the results. Therefore, the 
question can be asked: Do bilingual program directors report greater usage of TAAS data than 
any other type of data? The deductions of any findings for this question are problematic due to 
the nature of the Texas accountability system. Because the TAAS results are used to rate 
districts, district personnel might be more likely to use the data regardless of the format of the 
report. 

Methodology 

Design and Subjects 

Survey methodology following Dillman’s Tailored Design Method (2001) was utilized to 
achieve the purpose of the study. The study began in May 2000 with the electronic mailing of 
prenotice letters and ended in October 2000 with the final letter contact. Time of year was a 
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consideration in determining when to begin the study. The decision was made to contact 
participants at the end of the school year, when the likelihood of evaluation data availability 
would be the greatest. While some directors work through the summer, others leave for the 
summer break in early June. Ultimately, the survey was conducted through the summer and into 
the fall in order to achieve the necessary rate of participation. 

Subjects were identified from the 218 districts with bilingual education programs in Texas 
at the time of the study, Spring 2000. In order to have results that could be generalized to this 
population, a random sample of 140 subjects was needed (Krejcie & Morgan, 1970, p. 608-609). 
Subjects were sent the Bilingual Program Director Survey. Sixty-six percent or 93 subjects 
responded to the survey. Survey data were merged with district data provided by the Public 
Information Management System (PEIMS). 

Comparisons were made between respondents and nonrespondents using PEIMS data to 
assess the possibility of response bias. Specifically, the t test case II independent was performed 
using 1 1 student demographic variables (e.g., total student enrollment, LEP enrollment, number 
students tested, LEP exemption rate, percent students passing, and percent LEP passing). No 
significant differences between the respondents and the non-respondents on these variables were 
identified. Thus, it can be assumed that reponse bias possesses little threat to the external validity of 
the study’s findings. 

Instrumentation 

The Bilingual Program Director Survey is composed of four sections that pertain to the 
following areas: (1) accountability data, (2) student characteristics, (3) evaluation practices, and (4) 
.demographic information. The first section of the survey lists five types of accountability data 
required to be collected by Texas bilingual programs: oral language proficiency level of LEP 
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students, English norm-referenced testing, Spanish language norm-referenced testing, TAAS 
criterion-referenced testing, and LEP student attendance data. For each type of accountability data, 
there are six statements with response options of “yes” or “no.” The statements reflect a variety of 
uses that program directors could have for the data. The statements were: (a) This data are collected 
for individual students, (b) This data are aggregated for district-wide reports, (c) I am responsible 
for the collection of this data, (d) I analyze this data, (e) I receive a report on this data from another 
department in the district, and (f) I am responsible for reporting this data to decision-makers in my 
district. 

The second section of the survey pertaining to student characteristics has four questions. 

The first question includes the same five types of accountability data as in section one: oral 
language proficiency level of LEP students, English norm-referenced testing, Spanish language 
norm-referenced testing, TAAS criterion-referenced testing, and LEP student attendance data. 

Only this time program directors were asked to indicate their level of agreement with the 
statement, “1 believe this measurement provides valid information regarding the academic 
progress of limited English proficient students.” The second question in section two asked 
program directors to respond to the degree of impact for the following data: number of LEP 
students involved in special education, number of LEP students identified for gifted and/or 
talented programs, number of LEP students participating in extra-curricular activities, and 
number of secondary LEP students taking honors/ AP classes. The third question in section two 
asked participants to indicate the type of usage they have with each of seven types of 
accountability data: (1) oral language testing district-wide results, (2) English norm-referenced 
testing district-wide results, (3) Spanish language norm-referenced testing district-wide results, 

(4) TAAS district-wide LEP student results, (5) LEP student attendance data, (6) LEP student 
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mobility information, and (7) LEP student grade information (i.e., report card grades). The fourth 
question in section two asked participants to look at the same seven indicators as in the third 
question and select the response they believed to be most accurate. Subjects were given five 
choices: (1)1 believe this measurement provides valid information regarding the academic 
progress of limited English proficient students, (2) I have access to this data when I need it, (3) 
This data is reported to decision-makers in my district, (4) I believe this data is critical to the 
effective implementation of bilingual programs, and (5) I believe this data is taken into account 
in programmatic decisions made in my district. 

The third section of the survey is made up of six questions. Each question is intended to 
provide insight as to the dynamics of data collection, evaluation, and reporting in the program. 
The first question asked, “As a whole, how do you typically analyze student data?” The 
responses were: (1)1 analyze data for students who are participating in bilingual/ESL programs, 
(2) I analyze data for students who are not participating in bilingual/ESL programs, (3) I analyze 
both groups individually, (4) 1 analyze both groups comparatively, or (5) None of the above. The 
second question asked subjects to identify, if any, the barriers to having access to student data in 
decision-making. If the participant selected “yes,” he/she was asked to identify the greatest 
barrier. The third question asked subjects whether or not they believed that evaluation data are 
taken into account by decision-makers in their district. The fourth question asked program 
directors to estimate the percentage of their total program evaluation that is State-required 
student assessment data. The fifth question asked subjects to identify the philosophy of bilingual 
education that most closely approximated their own: the immersion model, the transitional 
model, and the maintenance model. The sixth question asked directors to identify who in the 
district was responsible for collecting the data listed above, if it was not collected by the director, 
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the question asked directors to identify which of the three philosophies of bilingual education 
most closely matched that of the person or department who collected the data. The final question 
in this section asked subjects whether or not they believed that student data are equally available 
for LEP students and non-LEP students. 

The fourth section of the Bilingual Program Director Survey asked subjects for 
demographic information. Questions of age, gender, ethnicity, highest degree held, certifications 
held, years experience in education, job title, and years in current position, were asked of 
directors. Program directors were also asked questions about their program. Specifically, they 
were asked about the number of administrators in the bilingual department, Title Vll grants, and 
student distribution in various programs. 

Each section of the instrument was developed from the literature. Participants from a national 
roundtable discussion group were asked to analyze the instrument for content validity. Instrument 
stability was established by examining overlapping questions within the survey. Cronbach’s alpha 
was calculated for each subsection of the survey and for all survey data but demographic 
information. An overall alpha coefficient of .96 was found, with each subsection possessing 
coefficients that ranged from .72 to .97. Nunnaly (1978) reports that coefficients .70 or higher 
indicate an acceptable level of internal consistency reliability. 

Data Analysis 

To address the problem and purpose of the study, analysis of responses focused on the 
following sections of the survey: (a) the types of data used for decision-making, (b) the types of 
decisions made, (c) evaluation activities conducted by bilingual departments, and (d) access to or 
availability of data. A descriptive analysis was conducted with the generation measures of central 
tendency and variablity for the types of data used for decision making, types of decisions made 
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with data, evaluation activities conducted by the bilingual department, and availability of data for 
decision making. Further, responses to questions of use for specific data types were combined to 
create a factor score, “level of usage”(LOU), for each of data type. These factor scores were then 
summed to generate an overall LOU score pertaining to directors’ use of accountability data. The 
highest possible overall LOU score was 139 and the lowest was 0. 

An overall availability of data score was also generated for the eleven data types for 
directors. A new variable was computed, which was labeled overall data accessability, with 
responses grouped into three levels of access to data: no restrictions, minor restrictions, and 
major restrictions. 

The descriptive analysis of the data provides the basis for further examination of the 
conduct factor. Specifically, the three questions that emerged from the review of literature on the 
conduct factor and Texas bilingual education were transformed into hypotheses. The first 
question identified in the review became the hypothesis: Bilingual program directors who believe 
in using the student’s native language for instruction will report higher levels of use of Spanish 
norm-referenced tests than bilingual program directors who believe in English immersion 
programs. To test this hypothesis, program directors were classified into categories: those who 
believe in using the students’ native language for instruction and those who do not. The question 
pertaining to the director’s philosophy of bilingual education was used for this determination. 
Next, the LOU score that was created for use of Spanish norm-referenced tests was defined as 
the dependent variable and the statistical procedure of t test case II independent was performed. 

The second question offered in the literature was transformed into the hypothesis: There 
is a relationship between the availability of data and usage of data as reported by bilingual 
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program directors. Spearman’s rho was calculated using the overall data accessability and the 
overall LOU. The assumptions of linearity for this test were examined using a scattergram. 

The third question pertaining to differences between use of TAAS and other student data 
was transformed into 10 hypotheses, each pertaining to testing differences in the LOU scores 
between TAAS and the other 10 student data sources. The statistical procedure of matched 
samples was used to determine whether bilingual program directors reported greater usage of 
TAAS data than any other type of data. 

Results 

The analysis of the data begins by examining the kinds of decisions and the type of 
student data bilingual program directors use for decision making. Following this examination the 
analysis of responses by the directors on the types of evaluation activities conducted in their 
districts will be presented. Availability of data is the third topic to be offered in this section. 
Finally, the analysis of the data for testing the conduct factor hypotheses will be offered. 
Following the results is a statement of conclusion and significance of the study. 

Decisions and Student Data 

There were nine decisions that were listed and were taken from the work of Alkin et al. 
(1974) and eleven types of student data on the Bilingual Program Director Survey. The 
responses by the bilingual education program directors to these questions were found to fall out 
into four categories when the percentages of directors responding to the decision by data 
categories were ranked. These decision categories were labeled: informal, formal, instructional, 
and non-instructional. 

One kind of decision, identify problems, composes the first category labeled as informal 
decision making. Identifying problems emerged as the most frequently engaged in decision 
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making process for which directors used student data. Specifically, 99% of directors reported 
using TAAS data for identifying problems and 89% used oral language data. These were the 
most frequently cited student data sources used for most of the nine decisions, however the use 
of TAAS and oral language was particularly high for identifying problems. 

Eighty-one percent of directors also reported using retention rates, 73% high school “on 
track to graduate,” 71% English norm-reference testing, and 71% attendance data for identifying 
problems. Such data uses communicate an attention the “flow” of students through the various 
programs, which culuminates in an exiting of students into the general education program and 
monitoring of the data to identify the potential threat of students dropping out of school. 

Hispanic students, while not the only receptients of bilingual services, compose a large 
percentage of the students who receive services and their “at-riskness” for dropping out is well 
known in the literature (Rumberger & Larson, 1998) and by practitioners. 

Student mobility was used to identify problems by 64% of directors, Spanish norm- 
referenced tests by 62%, report card grades by 57%, student discipline by 56%, and student SES 
by 52%. It is important to note that not all of these data are readily available to directors, a point 
that will be discussed in greater detail later. Availability may be influencing some of the use 
patterns presented here. However, the general picture that develops given the above responses is 
one of directors constantly, informally, vigilantly, always-in-the-back-of-their-minds attending to 
student data to assess or identify current or future problems that will need to be addressed; 
particularly highly charged public and political issues that revolve around accountability ratings 
and the acquisition of English, Spanish, or both languages for limited English proficient students. 
It is as if directors gage plans, actions, and events within a program, department, or the district 
according to impact — whether real or perceived — on these measures. And in particular, they 
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judge to be problems those incidents that may negatively impact student TAAS scores. 

Two of Alkin’s decision topics were found closely connected with each other and are 
associated with formal decision making activities by bilingual directors: preparing reports and 
recommending program changes. Table 1 titled Percent of Directors Who Selected Each Student 
Data Type for Formal Decisions , provides the data. Preparing reports and recommending 
program changes are formal procedures that are performed for entities outside the department 
such as district, state, and federal agencies. Such activities occur on a schedule estabilished by 
others and would include mandated information made by the report requesting entity. 

Table 1. 

Percent of Directors Who Selected Each Student Data Type for Formal Decisions 



Data type 


Reports 


Recommendations 


TAAS 


97% 


97% 


Oral Language Testing 


82% 


75% 


Retention Rates 


61% 


58% 


English Norm-Referenced Testing 


71% 


64% 


Spanish Norm-Referenced Testing 


63% 


55% 


High School Status of "On track to graduate" 


43% 


59% 


Student SES Data 


56% 


43% 


Attendance Data 


57% 


41% 


Student Mobility 


40% 


43% 


Student Discipline Data 


29% 


28% 


Report Card Grades 


37% 


38% 



Again, TAAS and oral language testing receive the highest frequency of responses. The 
frequency for each of the data sources for recommending program changes are slightly lower 
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than those for preparing reports, except for high school status “on track to graduate” and student 
mobility. What appears most interesting is the “low” use of oral language testing data for 
recommending program changes compared to TAAS data. It is also a drop from the level of its 
use compared to the informal, identifying problems, discussed earlier. 

Additionally, student discipline and report card grades are not used frequently in the 
formal decision making process. Student grades are different from norm-referenced data in that 
grades are a closer reflection of the nature of the student and teacher interaction. Teacher affect 
is perceived to influence these ratings, among other things, and thus student grades by teachers 
are considered less reliable than norm or criterion-reference data. Thus, program directors may 
not attend to this data because they do not feel that it is useful or that it is not important. Program 
directors may also not use this information because they do not have access to it. Whatever the 
reasons, discipline and grades, which are two measures from and about the context of the 
classroom, are noticeably ignored by program directors in their formal decision making 
processes. 

Making staff development changes, changing instructional delivery, and changing 
instructional materials emerged as a decision making area in the analysis of the frequency pattern 
of director use of student data. Table 2 presents this information. Unsurprisingly, testing data, 
including TAAS, oral language, English norm-referenced, and Spanish norm-reference data 
receive attention and less so other student data except for student SES. That 52% of directors 
reported using student SES in instructional decision making is understandable given Ruby 
Payne’s (1995) recent influence on Texas staff development. The frequency with which directors 
attend to student discipline and report cards is higher for instructional than for formal decisions. 
However, they are still at levels that could be considered low. 
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Of the eleven data sources, report card grades appeared to receive the lowest level of 
attention by directors. Given that grades are given to parents on a regular basis throughout the 
year, in some districts as frequently as every three weeks, but in most every six weeks, it would 
seem that they are considered of value. However the use pattern for directors would indicate that 
that they do not find this data useful, that they do not considered this data important, or they do 
not have access to this data. The contradiction of making such data available to parents for 
understanding student progress and learning from teacher instruction but not directors is 
revealing, as well as the other possible explanations for the identified use patterns. 

Table 2. 



Percent of Directors Who Selected Each Student Data Type for Instructional Decisions 



Data type 


Staff Develop. 


Instr. Delivery 


Instr. Materials 


TAAS 


96% 


94% 


98% 


Oral Language Testing 


73% 


77% 


77% 


Retention Rates 


60% 


53% 


58% 


English Norm-Referenced Testing 


62% 


63% 


64% 


Spanish Norm-Referenced Testing 


54% 


59% 


57% 


High School Status of "On track to graduate" 


43% 


61% 


54% 


Student SES Data 


52% 


38% 


34% 


Attendance Data 


33% 


33% 


25% 


Student Mobility 


33% 


38% 


27% 


Student Discipline Data 


49% 


32% 


28% 


Report Card Grades 


44% 


41% 


36% 



The remaining three decisions — changing community relations, changing program 
management, and making personnel changes — were found to share certain characteristics. These 
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three decision types were labeled non-instructional decisions and their data is presented in Table 
3. Several aspects of the data in the table stand out. Most noticeably, student academic 
performance data received significantly lower levels of responses for changing community 
relations (e.g., 51% TAAS, 26% oral language, and 21% and 22% for norm-referenced tests). 
Several explanations could account for these findings. First, public reporting of student 
performance and the community relations’ issues that follow could be considered in Texas 
districts the “jurisdiction” of others (i.e., school principals or the superintendent). Another 
explanation could be that directors do not feel that the parents of their students understand or 
desire to understand student performance data. Another explanation could be that directors are 
not completely comfortable with the data themselves and might not know how to go about 
explaning these data to parents. No matter the reason, a clear void in the connection is evident 
between the bilingual department leadership and parent/community constituents on this critical 
topic of student performance. 

Interestingly, responses by directors indicate that they attend in changing community 
relations to student data (i.e., attendance, mobility, student SES, and discipline) at levels 
comparable to TAAS. The attention to these data suggests that directors are attempting to 
understand the home lives of their students through such data. Given these findings the picture 
that develops is of limited or one-way attention to community relations. 

Limited attention also appears to be given to changing program management and making 
personnel change. Or rather the use of student data for these non-instruction decisions appears 
rather limited compared to usage levels for the other decision types previously discussed. It may 
be that other data (e.g., classroom observations, teacher evaluations, and other informal measures 
of morale) receive more attention and thus the lower response for student data. However, such 
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reasons suggest less attention or priority of student data for decision making on issues not 
directly connected to instruction. 

Table 3. 



Percent of Directors Who Selected Each Student Data Type for Non-Instructional Decisions 



Data type 


Community 


Management 


Personnel 


TAAS 


51% 


71% 


78% 


Oral Language Testing 


26% 


47% 


42% 


Retention Rates 


46% 


45% 


49% 


English Norm-Referenced Testing 


21% 


44% 


40% 


Spanish Norm-Referenced Testing 


22% 


39% 


38% 


High School Status of "On track to graduate" 


29% 


34% 


41% 


Student SES Data 


46% 


28% 


21% 


Attendance Data 


49% 


26% 


21% 


Student Mobility 


49% 


31% 


20% 


Student Discipline Data 


46% 


36% 


38% 


Report Card Grades 


34% 


15% 


24% 



A level of usage (LOU) score was calculated for each type of student data discussed 
above by combining the responses for all decision types. The LOU score for each type of student 
data is reflected in the ordering of the variables in Tables 1-3 and thus has already been 
discussed. These scores where then summed to calculate an overall LOU score. An average 
overall LOU score for participants of 73.44 (SD = 24.86) was found. The distribution of scores 
was normal. With a potential value range of 0 to 139, the average overall LOU score reveals that 
bilingual program directors generally do use data for decision-making. However, Tables 1-3 
demonstrated that some types of data are used more than others. 
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Evaluation Activities Conducted by Bilingual Departments 

Bilingual program directors were asked to report on the types of evaluation activities 
conducted in their districts. Specifically, directors were prompted to indicate whether or not 
certain statements about individual students, aggregated data for district-wide reports, 
responsibility for data collection, data analysis, and responsibility for reporting data to district 
decision-makers were true for the five types of data: oral language proficiency level of LEP 
students, English norm-referenced testing, Spanish norm-referenced testing, TAAS criterion- 
referenced testing for LEP students, and LEP high school attendance data. From the responses to 
these questions the following analysis emerged. 

All bilingual program directors (i.e., 100%) reported conducting oral language 
proficiency tests, 99% TAAS testing, 79% English norm-referenced testing, and 70% Spanish 
norm-referenced testing. When asked if they collect LEP high school student attendance data for 
individual students, 79% responded positively. 

All directors report conducting oral language proficiency testing, however, only 76% 
report the data are aggregated for district-wide reports. On the other hand, all those reporting 
TAAS testing is conducted for LEP students also report the data are aggregated for district-wide 
reports. It is questionable, however, that both English and Spanish LEP student TAAS results are 
aggregated for district-wide reports. At this time, the Texas Education Agency provides separate 
district wide reports for TAAS results in English and TAAS results in Spanish. This is 
particularly problematic for bilingual programs because many students are tested in English. Yet 
when TAAS scores are returned to districts, Spanish TAAS scores are often associated with the 
success or failure of bilingual education programs. Of the 79% of bilingual program directors 
who reported English norm-referenced testing is used in their districts, 82% report the data are 
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aggregated for district-wide reports. Similarly, 83% of those who report using native 
language/Spanish norm-referenced testing report the data are aggregated for district-wide 
reports. Of those who report LEP high school student attendance data are collected for individual 
students, 88% report the data are available in district-wide reports. With the exception of TAAS 
results, an average of 19% of bilingual program directors report that only individual student data 
are collected for these data types. No district-wide analysis can be conducted for students at 
certain grade levels or for students with various amounts of time in the program. 

In response to questions pertaining to the collection of data, 77% report they are 
responsible for collecting oral language proficiency testing data, 58% Spanish norm-referenced 
testing, 52% English norm-referenced testing data, 46% TAAS testing data, and 20% LEP high 
school student attendance data. 

When asked if they are responsible for analyzing the types of data, 86% of the bilingual 
program directors report they analyze TAAS data, 85% oral language proficiency testing data, 
83% Spanish norm-referenced testing, 79% English norm-referenced testing data, and 53% LEP 
high school student attendance data. 

Directors were asked a further question about how they analyze student data. Of those 
who analyze data, 54% of directors reported that they most often analyze student data by 
comparing data for students who are participating in bilingual/ESL programs with students who 
are not participating. The second most frequent type of analysis is analyzing data only for 
students who are participating in a program reported by 29% of directors. Twelve percent 
reported they analyze both groups of students individually, and one percent reported they analyze 
only data for students who are not participating. 

When asked if they are responsible for reporting data to decision-makers in their districts, 
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77% of the bilingual program directors indicate they report oral language proficiency testing data 
to decision-makers, 69% report Spanish norm-referenced testing, 67% report TAAS data, 60% 
report English norm-referenced testing data, and 33% report LEP high school student attendance 
data. It is interesting to note that the data most highly valued among districts — TAAS data — are 
not frequently reported to decision-makers by the bilingual program director. Because the 
directors report elsewhere that TAAS data are reported to decision-makers, it is supposed that 
another person/department is responsible for reporting LEP TAAS data to decision-makers in 
31% of the districts. 

Elsewhere, program directors were asked about the impact of various reports. Fourteen 
percent of the bilingual program directors reported they have seen a report regarding the number 
of LEP students involved in special education but that it did not have a major impact on them. 
Similarly, 13% reported the same for the number of LEP students identified for gifted and/or 
talented programs and 13% for the number of LEP students participating in extra-curricular 
activities. Only 7% reported they had seen a report on the number of secondary LEP students 
taking honors/AP classes but again that the data did not have a major impact on them. Such 
responses lead into the responses by directors about the availability of data for decision making. 
Availability of Data. 

The following percentages of program directors report data are available when they need 
it: TAAS by 97%, oral language testing by 95%, retention rates by 85%, attendance data by 82%, 
high school status of “on track to graduate” by 81%, student socio-economic status data by 79%, 
English norm-referenced testing data by 78%, student discipline data by 73%, Spanish norm- 
referenced testing data by 71%, student mobility information by 68%, and report card grades by 
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65% of directors. Thus, only 3% reported they do not have access to TAAS data when they need 
it, while 35% claim they do not have access to report card grades when needed. 

When the overall data accessibility was calculated by summing each directors responses 
to these nine questions a clearer picture of director accessibility to student data is gained. Thirty 
seven percent of directors reported unrestricted accessability to all data, while 29% reports some 
restrictions to data, and 34% reported major problems with accessing student data. Given that 
this data “should” be available to directors, the label of major was applied when directors 
reported being unable to access a third or more of the data for their students. 

In triangulating responses, directors were also asked whether or not they experienced 
barriers in having access to student data for decision-making. Not having enough staff was 
identified by 27% of directors. Others reported not having enough time by 16%, and a lack of 
evaluation knowledge and skills by 9% of directors. 

Finally, directors were asked to indicate whether or not student data are equally available 
for LEP students and non-LEP students. Seventy-four percent of the program directors reported 
student data are equally available for LEP students and non-LEP students. Nineteen percent 
(almost one-fifth) reported non-LEP student data are more available than LEP student data, and 
four percent reported LEP student data are more available than non-LEP student data. 

Conduct Factor 

The conduct factor questions and resulting hypotheses that emerged from the literature 
were tested to enhance the understanding about evaluation use of bilingual directors. 

Specifically, the data was analyzed to test the hypothesis: Bilingual program directors who are 
more committed to using the student’s native language for instruction will report higher levels of 
use of Spanish norm-referenced tests than bilingual program directors who are less committed to 
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using the student’s native language for instruction. Seventy-one percent of directors were found 
to embrace a maintenance model (i.e., students should receive instruction in both languages so 
that they can be bilingual), while the remaining 28% embraced an immersion or transitional 
model. A t-test case II for independent means was performed and resulted in a t (78, -2.274 = 
.026 (2-tailed), which is statistically significant at the .05 alpha level. The mean LOU for 
Spanish norm-reference testing for directors who embraced the maintenance model was 5 ( SD = 
3.4); on a scale of one to seven. For directors who embraced the other two models the mean LOU 
was 3 {SD = 3.4). This finding supports the conclusion that beliefs in using Spanish for 
instruction by program directors are associated with higher levels of usage of Spanish norm- 
referenced testing data. 

The second hypothesis that stated, there is a relationship between the availability of data 
and usage of data as reported by bilingual program director, was tested using Spearman’s rho. 
This correlation coefficient was calculated to be 0.49 for the variables overall data accessability 
and overall LOU. Therefore, slightly less than 25% of the variance in overall LOU is shared with 
overall data accessability. These results support the conclusion that when directors have access to 
data when needed, they use the data for decision-making. However, such findings also support 
the argument that when directors perceive that they need data, they gain access to the data. 

The final component of the conduct factor to be examined pertains to the level of use 
given the familiarity of the evaluation. It was hypothesized that given the implimentation and 
high visability of TAAS, program directors would report significantly higher LOU scores for this 
data than for any of the other student data sources. A one-sample t-test was performed once 
using the directors’ average LOU score for TAAS data as the constant. The LOU scores for the 
other 10 ten data types were compared with the mean LOU score for TAAS data. 
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The average LOU score for TAAS data was 7.83 ( SD = 1.47). Other data types had the following 
average LOU scores: oral language testing 5.93 (SD = 2.57), English norm-referenced testing 
5.05 (SD = 3.30), Spanish norm-referenced testing 4.53 (SD = 3.53), attendance data 3.54 (SD = 
2.72), report card grades 3.25 (SD = 2.70), student mobility information 3.52 (SD = 2.74), high 
school status of “on track to graduate” 4.43 (SD = 3.04), student discipline data 3.40 (SD = 2.87), 
student SES data 3.84 (SD = 3.03), and retention rates 5.17 (SD = 3.07). As displayed in Table 4, 
the results of one-sample t-tests comparing the LOU score for each type of data with the LOU of 
TAAS data resulted in t-scores all significant at the .001 alpha level. The results led to the 
conclusion that bilingual program directors report significantly greater usage of TAAS data than 
any other type of data. 



Table 4. 

One Sample t Test Comparing LOU of TAAS Data and Other Data Types 



Data Type 


df 


/ 


Significance (2-t) 


Oral language testing 


84 


-6.81 


.000** 


English norm-referenced testing 


87 


- 7.92 


.000** 


Spanish norm-referenced testing 


80 


-8.42 


.000** 


Attendance data 


87 


-14.97 


.000** 


Report card grades 


15 


-6.80 


.000** 


Student mobility information 


86 


-14.66 


.000** 


High school status of “on track to graduate” 


90 


-10.66 


.000** 


Student discipline data 


88 


-14.54 


.000** 


Student SES data 


86 


-12.27 


.000** 


Retention rates 


86 


-8.07 


.000** 



**p<. 001. 




27 



Accountability Data 27 



Conclusions 

Texas bilingual program directors reported that they do use data to improve education 
practices within their school districts. Unfortunately, they do not utilize the wealth of data collected 
for reports to the State, but concentrate on a singular component of student data: TAAS scores. This 
raises serious issues about the comprehensiveness of program decision-making. Considering that the 
majority of Texas bilingual programs are transitional bilingual education program models and that this 
model calls for students to transition to all-English instruction by the 3 rd or 4 th grade, it seems 
incongruous that the primary source of data used is data that is not collected for students until they are 
in the 3 rd grade. Presumably this would be very late to begin using student data to measure student 
progress. Earlier test data would seem critical to the administration of Texas bilingual programs. In 
fact, due to additional testing requirements for LEP students, bilingual directors could potentially 
access annual oral language testing data, English norm-referenced testing data, and native language 
norm-referenced testing data from the time students enter the district. But Texas bilingual education 
directors do not appear to use these data sources for instructional decision-making. At first this may 
seem illogical, but a closer examination of district practices reveals that although it is not a sound 
educational practice, the valuing of TAAS data over any other data type may actually be quite rational. 

TAAS data are the only assessment data that are used by the State to determine school 
ratings {Low Performing, Acceptable, Recognized, or Exemplary). School ratings affect annual 
personnel evaluations for district administrators, campus principals, and teachers. Ratings are 
published in newspapers and magazines. In reporting their areas of responsibility, bilingual program 
directors overwhelmingly reported their greatest area of responsibility is ensuring their districts are 
following state and federal guidelines. Because of the emphasis places on TAAS data by the state, it 
is reasonable that bilingual program directors reported great attention to TAAS data for informal, 
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formal, and instructional decision-making. But this does not fully explain a lack of attention to 
other types of data. 

A study of the evaluation activities conducted by bilingual departments reveals several 
important insights that may partly explain such findings. First, some types of data are more 
widely collected than others. While oral language proficiency testing and TAAS testing is 
conducted among virtually all bilingual programs, there is greater variance among districts 
regarding the other types of data collected. Secondly, much of the data are not reported to 
decision makers by the bilingual program director. In many cases the data are reported by 
another person/department in the district or not reported at all. Third, some types of data are 
more available than others. While TAAS and oral language testing data are widely available, 
other types of data are not available when needed in 1 5% - 35% of the cases. Fourthly, there are 
barriers to accessing student data including not having enough staff, not having enough time, and 
a lack of evaluation knowledge and skills. In one out of five cases, non-LEP student data are 
more available than LEP student data. Fifthly, state-required assessment is a primary source of 
evaluation data among bilingual programs. Finally, while some types of data are more widely 
used than others, bilingual program directors were found to generally use data for decision- 
making. The average program director reported using data on more than half of the questions on 
the survey. 

The examination of the hypotheses revealed the importance of three components of the 
conduct factor. First, a strong belief in using Spanish for instruction was found associated with 
higher levels of usage of Spanish norm-referenced testing data. Second, a correlation was found 
to exist on data usage and data availability. These results lead to the conclusion that the 
availability of data has an important relationship with the use of data. Third, the results of the last 
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hypotheses showed that bilingual program directors did report significantly greater usage of 
TAAS data than any other type of data. The emphasis on TAAS data usage is tremendous, 
especially in light of the lack of use of other types of student data. 

The two primary goals of the AEIS and PEIMS programs in Texas are to measure student 
progress toward achieving academic excellence and to improve education practices of local 
school districts. These systems require districts to conduct state-mandated testing and to submit a 
variety of student data such as attendance, dropout, and graduation rates, ethnicity, SES, gender, 
mobility rates, and student discipline data. Evidence from Texas bilingual program directors 
indicates that Texas educators do, in fact, use student data to measure student progress toward 
achieving academic excellence and to improve education practices within their school districts. 
Unfortunately, they do not access and utilize the wealth of data collected for reports to the State, 
but concentrate instead on a singular component of student data: TAAS scores. This raises 
serious concern that bilingual directors do not appear to value a wide range of data for decision- 
making and therefore do not have a comprehensive perspective of the academic achievement of 
students in their programs. 

However, the results of the study show that this situation can be changed, but in order to 
do so data must be made accessible to educational decision-makers. Bilingual program directors 
do use assessment data for decision-making. The scope of this study was limited to bilingual 
program directors, but certainly similar issues exist among other educational leaders. The issue 
of data availability must be approached at multiple levels. 

The State has a role in the availability of data. As indicated in this study, Texas educators 
use data that comes to them in familiar formats. The state provides a wealth of TAAS data to 
educators across the State. The State disaggregates data by student gender, ethnicity, SES, 
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campus, grade level, and special program participation. But other data types are not as widely 
available. The findings of this study suggest that Texas educators would be likely to use data 
such as student SES, student attendance rates, student mobility rates, and student discipline data 
if the data were available. The State has begun a noble quest to monitor student achievement and 
improve educational programs. Funding and systems are in place to develop, conduct, score, and 
report TAAS testing. Additional efforts must be made to provide similar funding and systematic 
support in order that other types of student data are equally available. 

Districts have a role in making data available to decision makers within the district. The 
superintendent must realize the importance of data availability in measuring the academic 
progress of students and improving instruction programs. Until the State makes data more 
accessible, districts will likely have to assume responsibility for doing so. A key individual in the 
district who is responsible for data dissemination could facilitate this task. This key individual 
would have the authority to access all Public Education Information Management System 
(PEIMS) and Academic Excellence Indicator System (AEIS) data, have the data available in 
electronic formats (text files) that could be analyzed through tools such as SPSS, and disseminate 
data to decision makers within each department and campus. As long as there is not one single 
point of contact for data flowing to and from the state, the data will be fragmented. 

Finally, bilingual education leaders must assume responsibility in accessing and utilizing 
a wide variety of student data for measuring academic achievement and improving instructional 
programs. Many reasons for non-use may be overcome when one person in the organization 
becomes an advocate for data use. This person contextualizes information by interpreting the 
data in terms of the current situation and by defining what the data mean to decisions at hand. 
Alkin (1985) found that many times the most appropriate person to advocate for the use of data 
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is the program director. In his studies of bilingual education, program directors were the key to 
data usage. If they validated the information and demonstrated they would use the data, other 
educators within the organization also used the data for decision-making, thus monitoring and 
improving bilingual education programs in their districts. Lam (1992) asserts, “The controversial 
nature of bilingual education is partly due to political squabbling and partly due to the inadequate 
practices in the evaluation of the implementation and effects of bilingual education” (p. 183). 
Collier and Thomas, (1998), emphasize the importance of assessment and evaluation among 
teachers and leaders of educational programs for limited English proficient students. “We 
educators — teachers and administrators — must understand assessment, keep informed of the 
constantly changing policy framework, and use assessment well to inform our practice” (p. 240). 
With the huge commitment of resources to conduct assessment and gather data on Texas school 
children, it is imperative that bilingual directors use data to monitor and improve the variety of 
bilingual program models, particularly at this critical time in educational history when external 
student academic achievement requirements are being imposed in a highly political climate. 
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